Subtle Social Behavior through Sensor Networks

With seed support (no Co-PI salary nor Ph. D student) from an NSF-funded Engineering Research Center (ERC), we have collected multimodal data—audio, video, Inertial Measurement Units (IMUs), and motion capture data—of individuals working in a kitchen [1]. To record human behavior in settings that are as natural as possible, we have installed a fully operational kitchen and captured 40 subjects cooking five different meals (brownies, salad, pizza, sandwich, and scrambled eggs) from start to finish. We recorded video from five stationary cameras and one wearable camera, audio from five balanced microphones, motion capture, and five IMUs. Figure 1 shows the IMUs, motion capture data and wearable camera setup.